A Throughput Driven Task Scheduler for Batch Jobs in Shared MapReduce Environments

MapReduce is one of the most popular parallel data processing systems, and it has been widely used in many fields. As one of the most important techniques in MapReduce, task scheduling strategy is directly related to the system performance. However, in multi-user shared MapReduce environments, the existing task scheduling algorithms cannot provide high system throughput when processing batch jobs. Therefore, in this paper, a novel scheduling technique, Throughput-Driven task scheduling algorithm (TD scheduler) is proposed. Firstly, based on the characteristics of shared MapReduce environments, we propose the framework of TD scheduler. Secondly, we classify the jobs into six states. Jobs in different states have different scheduling priorities. We also give the rules of state conversion, which can ensure the fairness of resource allocation and avoid wasting system resources. Thirdly, we design the detailed strategies for job selection and task assignment. The strategies can effectively improve the ratio of local task assignment and avoid hotspots. Fourthly, we show that our TD scheduler can be applied to the heterogeneous MapReduce cluster with small modifications. Finally, the performance of TD scheduler is verified through plenty of simulation experiments. The experimental results show that our proposed TD scheduler can effectively improve the system throughput for batch jobs in shared MapReduce environments.